我们为模仿学习提供了一个新的框架 - 将模仿视为政策和奖励之间的基于两人排名的游戏。在这个游戏中，奖励代理商学会了满足行为之间的成对性能排名，而政策代理人则学会最大程度地提高这种奖励。在模仿学习中，很难获得近乎最佳的专家数据，即使在无限数据的限制下，也不能像偏好一样对轨迹进行总订购。另一方面，仅从偏好中学习就具有挑战性，因为需要大量偏好来推断高维奖励功能，尽管偏好数据通常比专家演示更容易收集。经典的逆增强学习（IRL）的配方从专家演示中学习，但没有提供从离线偏好中纳入学习的机制，反之亦然。我们将提出的排名游戏框架实例化，并具有新颖的排名损失，从而使算法可以同时从专家演示和偏好中学习，从而获得两种方式的优势。我们的实验表明，所提出的方法可实现最新的样本效率，并可以从观察（LFO）设置中学习以前无法解决的任务。

许多连续的决策问题是使用使用其他一些策略收集的历史数据，需要使用历史数据的高赌注并要求新策略（OPE）。提供无偏估计的最常见的OPE技术之一是基于轨迹的重要性采样（是）。但是，由于轨迹的高方差是估计，最近通过了基于国家行动探索分布（SIS）的重要性采样方法。不幸的是，虽然SIS经常为长视野提供较低的方差估计，但估算状态行动分配比可能是具有挑战性的并且导致偏差估计。在本文中，我们对该偏差差异进行了新的视角，并显示了存在终点是SIS的估计频谱的存在。此外，我们还建立了这些估算器的双重强大和加权版本的频谱。我们提供了经验证据，即该频谱中的估计值可用于在IS和SIS的偏差和方差之间进行折衷，并且可以实现比两者和SIS更低的平均平方误差。

面对顺序决策问题时，能够预测如果使用新策略进行决策会发生什么会发生什么。这些预测通常必须基于在一些先前使用的决策规则下收集的数据。许多以前的方法使得这种违规（或反事实）估计的性能测量值的预期值称为返回。在本文中，我们采取了迈向普遍违规估算机（UNO）的第一步 - 为返回分配的任何参数提供截止政策估计和高信任界限。我们使用UNO来估计和同时限制均值，方差，量级/中位数，分位式范围，CVAR和返回的整个累积分布。最后，我们还在各种环境中讨论了UNO的适用性，包括完全可观察，部分可观察的（即，与未观察到的混乱），马尔可夫，非马尔可瓦尔，静止，平稳的非稳定性和离散分布转移。

已知人类凝视是在操纵任务期间的潜在人类意图和目标的强大指标。这项工作研究人类教师的凝视模式证明了机器人的任务，并提出了这种模式可用于增强机器人学习的方式。使用Kinesthetic教学和视频演示，我们在教学中识别新颖的意图揭示凝视行为。这些在各种问题中被证明是从参考帧推理到多步任务的分割的各种问题。基于我们的研究结果，我们提出了两个概念验证算法，该算法表明，凝视数据可以增强多台任务的子任务分类，高达6％，奖励推理和策略学习，可为单步任务高达67％。我们的调查结果为机器人学习中的自然人凝视模型提供了基础，从演示设置上学习，并在利用人凝游来提高机器人学习的开放问题。

Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data, scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). This paper presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

Algorithms that involve both forecasting and optimization are at the core of solutions to many difficult real-world problems, such as in supply chains (inventory optimization), traffic, and in the transition towards carbon-free energy generation in battery/load/production scheduling in sustainable energy systems. Typically, in these scenarios we want to solve an optimization problem that depends on unknown future values, which therefore need to be forecast. As both forecasting and optimization are difficult problems in their own right, relatively few research has been done in this area. This paper presents the findings of the ``IEEE-CIS Technical Challenge on Predict+Optimize for Renewable Energy Scheduling," held in 2021. We present a comparison and evaluation of the seven highest-ranked solutions in the competition, to provide researchers with a benchmark problem and to establish the state of the art for this benchmark, with the aim to foster and facilitate research in this area. The competition used data from the Monash Microgrid, as well as weather data and energy market data. It then focused on two main challenges: forecasting renewable energy production and demand, and obtaining an optimal schedule for the activities (lectures) and on-site batteries that lead to the lowest cost of energy. The most accurate forecasts were obtained by gradient-boosted tree and random forest models, and optimization was mostly performed using mixed integer linear and quadratic programming. The winning method predicted different scenarios and optimized over all scenarios jointly using a sample average approximation method.

Lack of factual correctness is an issue that still plagues state-of-the-art summarization systems despite their impressive progress on generating seemingly fluent summaries. In this paper, we show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. To that end, we leverage information-theoretic measures of causal effects to quantify the amount of confounding and precisely quantify how they affect the summarization performance. Based on insights derived from our theoretical results, we design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Crucially, we give a principled characterization of data distributions where such confounding can be large thereby necessitating the use of human annotated relevant sentences to generate factual summaries. Our approach improves faithfulness scores by 20\% over strong baselines on AnswerSumm \citep{fabbri2021answersumm}, a conversation summarization dataset where lack of faithfulness is a significant issue due to the subjective nature of the task. Our best method achieves the highest faithfulness score while also achieving state-of-the-art results on standard metrics like ROUGE and METEOR. We corroborate these improvements through human evaluation.

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.

This paper describes Waymo's Collision Avoidance Testing (CAT) methodology: a scenario-based testing method that evaluates the safety of the Waymo Driver Automated Driving Systems' (ADS) intended functionality in conflict situations initiated by other road users that require urgent evasive maneuvers. Because SAE Level 4 ADS are responsible for the dynamic driving task (DDT), when engaged, without immediate human intervention, evaluating a Level 4 ADS using scenario-based testing is difficult due to the potentially infinite number of operational scenarios in which hazardous situations may unfold. To that end, in this paper we first describe the safety test objectives for the CAT methodology, including the collision and serious injury metrics and the reference behavior model representing a non-impaired eyes on conflict human driver used to form an acceptance criterion. Afterward, we introduce the process for identifying potentially hazardous situations from a combination of human data, ADS testing data, and expert knowledge about the product design and associated Operational Design Domain (ODD). The test allocation and execution strategy is presented next, which exclusively utilize simulations constructed from sensor data collected on a test track, real-world driving, or from simulated sensor data. The paper concludes with the presentation of results from applying CAT to the fully autonomous ride-hailing service that Waymo operates in San Francisco, California and Phoenix, Arizona. The iterative nature of scenario identification, combined with over ten years of experience of on-road testing, results in a scenario database that converges to a representative set of responder role scenarios for a given ODD. Using Waymo's virtual test platform, which is calibrated to data collected as part of many years of ADS development, the CAT methodology provides a robust and scalable safety evaluation.

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.